feat(e2e): add Python governance scenarios and CI workflow by prayagupa · Pull Request #3345 · microsoft/agent-governance-toolkit

prayagupa · 2026-07-15T23:51:58Z

Description

Adds a Python end-to-end governance suite covering five production control boundaries:

Policy denial prevents an unauthorized live budget mutation.
Policy allowance permits a non-diagnostic record update.
PII redaction removes sensitive data before model and tool boundaries.
Prompt-injection detection blocks poisoned retrieval content before model invocation.
Filesystem sandboxing prevents writes outside the configured workspace.

Adds a GitHub Actions workflow that runs the suite against Ollama and llama3.1. The workflow pins the Ollama release, verifies its archive checksum and model digest, caches the model, and uploads scenario results, JUnit output, and runtime metadata.

Also updates the v4 ratchet inventory, secret-scanning fixture allowlist, spell-check terms, and E2E documentation.

See example: https://github.com/microsoft/agent-governance-toolkit/actions/runs/29459498584/job/87499812340

Type of Change

Bug fix (non-breaking change that fixes an issue)
New feature (non-breaking change that adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation update
Maintenance (dependency updates, CI/CD, refactoring)
Security fix

Package(s) Affected

Checklist

My code follows the project style guidelines (ruff check)
I have added tests that prove my fix/feature works
All new and existing tests pass (pytest)
I have updated documentation as needed
I have signed the Microsoft CLA

Attribution & Prior Art

This contribution does not contain code copied or derived from other projects without attribution
Any external projects that inspired this design are credited in code comments or documentation
If this PR implements functionality similar to an existing open-source project, I have listed it below

Prior art / related projects (if any):

None.

AI Assistance

I can explain every meaningful change in this PR: what it does, why, and what tradeoffs were considered
I have run tests and verification appropriate for this change
No part of this PR was autonomously submitted by an AI agent without my review
I have not used AI to generate review comments on others' PRs

If AI tools materially shaped this change, briefly note what was used:

GitHub Copilot assisted with implementation, documentation, and validation. The contributor directed and reviewed the resulting changes.

IP, Patents, and Licensing

This contribution does not implement patent-pending or patent-encumbered techniques
This contribution does not require an NDA or licensing agreement to understand or use
Any AI tools used have terms compatible with the MIT License

Related Issues

None.

github-actions · 2026-07-15T23:52:31Z

PR Review Summary

Check	Status	Details
🔍 Code Review	⚠️ Missing	No current-run comment
🛡️ Security Scan	⚠️ Missing	No current-run comment
🔄 Breaking Changes	⚠️ Missing	No current-run comment
📝 Docs Sync	⚠️ Missing	No current-run comment
🧪 Test Coverage	⚠️ Missing	No current-run comment

Verdict: ⚠️ AI review incomplete; ready for human review

AI review comments are untrusted advisory output. The summary reports workflow-generated completion status only, not model-authored pass/fail claims.

github-actions · 2026-07-15T23:52:38Z

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

OpenSSF Scorecard

Package

Version

Score

Details

actions/actions/cache

55cc8345863c7cc4c66a329aec7e433d2d1c52a9

🟢 6.4

Details

Check	Score	Reason
Code-Review	🟢 10	all changesets reviewed
Maintained	🟢 10	13 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 10
Packaging	⚠️ -1	packaging workflow not detected
Binary-Artifacts	🟢 10	no binaries found in the repo
Dangerous-Workflow	🟢 10	no dangerous workflow patterns detected
CII-Best-Practices	⚠️ 0	no effort to earn an OpenSSF best practices badge detected
Token-Permissions	⚠️ 0	detected GitHub workflow tokens with excessive permissions
Pinned-Dependencies	🟢 3	dependency not pinned by hash detected -- score normalized to 3
Fuzzing	⚠️ 0	project is not fuzzed
License	🟢 10	license file detected
Signed-Releases	⚠️ -1	no releases found
Security-Policy	🟢 9	security policy file detected
SAST	🟢 10	SAST tool is run on all commits
Branch-Protection	⚠️ 1	branch protection is not maximal on development and all release branches

actions/actions/checkout

df4cb1c069e1874edd31b4311f1884172cec0e10

🟢 6.9

Details

Check	Score	Reason
Maintained	🟢 10	24 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 10
Code-Review	🟢 10	all changesets reviewed
Dangerous-Workflow	🟢 10	no dangerous workflow patterns detected
Binary-Artifacts	🟢 10	no binaries found in the repo
Token-Permissions	⚠️ 0	detected GitHub workflow tokens with excessive permissions
CII-Best-Practices	⚠️ 0	no effort to earn an OpenSSF best practices badge detected
License	🟢 10	license file detected
Fuzzing	⚠️ 0	project is not fuzzed
Packaging	⚠️ -1	packaging workflow not detected
Pinned-Dependencies	🟢 3	dependency not pinned by hash detected -- score normalized to 3
Signed-Releases	⚠️ -1	no releases found
Security-Policy	🟢 9	security policy file detected
SAST	🟢 10	SAST tool is run on all commits
Branch-Protection	🟢 5	branch protection is not maximal on development and all release branches

actions/actions/setup-python

a309ff8b426b58ec0e2a45f0f869d46889d02405

🟢 6.6

Details

Check	Score	Reason
Code-Review	🟢 10	all changesets reviewed
Maintained	🟢 10	17 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 10
Binary-Artifacts	🟢 10	no binaries found in the repo
CII-Best-Practices	⚠️ 0	no effort to earn an OpenSSF best practices badge detected
Packaging	⚠️ -1	packaging workflow not detected
Dangerous-Workflow	🟢 10	no dangerous workflow patterns detected
Pinned-Dependencies	🟢 7	dependency not pinned by hash detected -- score normalized to 7
Token-Permissions	⚠️ 0	detected GitHub workflow tokens with excessive permissions
Fuzzing	⚠️ 0	project is not fuzzed
License	🟢 10	license file detected
Signed-Releases	⚠️ -1	no releases found
Security-Policy	🟢 9	security policy file detected
Branch-Protection	⚠️ 0	branch protection not enabled on development/release branches
SAST	🟢 9	SAST tool is not run on all commits -- score normalized to 9

actions/actions/upload-artifact

043fb46d1a93c77aae656e7c1c64a875d1fc6a0a

🟢 5.2

Details

Check	Score	Reason
Code-Review	🟢 10	all changesets reviewed
Maintained	⚠️ 0	0 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 0
Binary-Artifacts	🟢 10	no binaries found in the repo
Packaging	⚠️ -1	packaging workflow not detected
Dangerous-Workflow	🟢 10	no dangerous workflow patterns detected
CII-Best-Practices	⚠️ 0	no effort to earn an OpenSSF best practices badge detected
Token-Permissions	⚠️ 0	detected GitHub workflow tokens with excessive permissions
Pinned-Dependencies	⚠️ 1	dependency not pinned by hash detected -- score normalized to 1
Fuzzing	⚠️ 0	project is not fuzzed
License	🟢 10	license file detected
Signed-Releases	⚠️ -1	no releases found
Security-Policy	🟢 9	security policy file detected
SAST	🟢 10	SAST tool is run on all commits
Branch-Protection	⚠️ 0	branch protection not enabled on development/release branches

actions/dtolnay/rust-toolchain

29eef336d9b2848a0b548edc03f92a220660cdb8

🟢 4.7

Details

Check	Score	Reason
Code-Review	⚠️ 0	Found 0/20 approved changesets -- score normalized to 0
Packaging	⚠️ -1	packaging workflow not detected
Maintained	⚠️ 2	3 commit(s) and 0 issue activity found in the last 90 days -- score normalized to 2
Token-Permissions	🟢 10	GitHub workflow tokens follow principle of least privilege
Pinned-Dependencies	⚠️ 0	dependency not pinned by hash detected -- score normalized to 0
Binary-Artifacts	🟢 10	no binaries found in the repo
Dangerous-Workflow	🟢 10	no dangerous workflow patterns detected
CII-Best-Practices	⚠️ 0	no effort to earn an OpenSSF best practices badge detected
Fuzzing	⚠️ 0	project is not fuzzed
License	🟢 10	license file detected
Signed-Releases	⚠️ -1	no releases found
Branch-Protection	⚠️ -1	internal error: error during branchesHandler.setup: internal error: some github tokens can't read classic branch protection rules: https://github.com/ossf/scorecard-action/blob/main/docs/authentication/fine-grained-auth-token.md
Security-Policy	🟢 3	security policy file detected
SAST	⚠️ 0	SAST tool is not run on all commits -- score normalized to 0

Scanned Files

.github/workflows/e2e-python.yml

github-actions · 2026-07-16T00:04:15Z

🤖 AI Agent: breaking-change-detector — API Compatibility

AI-generated review output. Treat it as untrusted analysis and verify before acting.

API Compatibility

No breaking changes detected.

github-actions · 2026-07-16T00:04:18Z

🤖 AI Agent: docs-sync-checker — Docs Sync

AI-generated review output. Treat it as untrusted analysis and verify before acting.

Docs Sync

tests/e2e_python/README.md -- newly added file, documentation is in sync.
.github/workflows/e2e-python.yml -- new workflow added, no documentation update required in README.md.
No new public APIs without docstrings detected.
CHANGELOG is missing an entry for the new Python governance scenarios and CI workflow.

github-actions · 2026-07-16T00:04:18Z

🤖 AI Agent: security-scanner — View details

AI-generated review output. Treat it as untrusted analysis and verify before acting.

No security issues found.

github-actions · 2026-07-16T00:04:21Z

🤖 AI Agent: test-generator — `tests/e2e_python/scenarios/policy_deny/test_policy_deny.py`

AI-generated review output. Treat it as untrusted analysis and verify before acting.

`tests/e2e_python/scenarios/policy_deny/test_policy_deny.py`

test_policy_deny_unauthorized_action -- Validate denial of unauthorized live budget mutation.
test_policy_deny_boundary_conditions -- Test edge cases for policy denial logic.

`tests/e2e_python/scenarios/policy_allow/test_policy_allow.py`

test_policy_allow_valid_action -- Ensure allowance of valid non-diagnostic record updates.
test_policy_allow_invalid_action -- Verify rejection of invalid actions under the policy.

`tests/e2e_python/scenarios/pii_redaction/test_pii_redaction.py`

test_pii_redaction_completeness -- Check that all sensitive data is redacted before model/tool boundaries.
test_pii_redaction_edge_cases -- Validate redaction for edge-case inputs (e.g., nested PII).

`tests/e2e_python/scenarios/prompt_injection/test_prompt_injection.py`

test_prompt_injection_detection -- Ensure poisoned content is detected and blocked.
test_prompt_injection_false_positive -- Verify no false positives for benign content.

`tests/e2e_python/scenarios/filesystem_escape/test_filesystem_escape.py`

test_filesystem_escape_prevention -- Confirm sandbox prevents unauthorized filesystem writes.
test_filesystem_escape_edge_cases -- Test sandbox behavior with edge-case paths or filenames.

github-actions · 2026-07-16T00:15:32Z

🤖 AI Agent: code-reviewer — View details

AI-generated review output. Treat it as untrusted analysis and verify before acting.

TL;DR: 0 blockers, 2 warnings. Comprehensive and secure implementation with minor follow-up considerations.

#	Sev	Issue	Where
1	W	Lack of explicit timeout for Ollama server	`.github/workflows/e2e-python.yml`
2	W	No validation for `AGT_E2E_MODEL_ATTEMPTS`	`.github/workflows/e2e-python.yml`

Action items:

Add an explicit timeout mechanism for the Ollama server startup to avoid indefinite hangs.
Validate AGT_E2E_MODEL_ATTEMPTS to ensure it is a positive integer. Fine as follow-up PRs.

MohammadHaroonAbuomar

.github/workflows/e2e-python.yml: the new uses: dtolnay/rust-toolchain@stable step is a mutable tag; this repo pins that exact action to dtolnay/rust-toolchain@29eef33 # stable in policy-engine-ci.yml (x2) and ci.yml:1074. SHA-pin it to match repo convention. Everything else in the ACS migration is verified sound (semantics preserved with default-deny strengthening; no fail-open at harness/SDK/Rust-core layers; ratchet exemption removal correct, ratchet passes locally; cache/upload-artifact bumps SHA-verified against upstream tags).

Signed-off-by: Prayag Upadhyay <prayag.upd@gmail.com>

Use an explicit protocol stub and a distinct canary value for the blocked filesystem write. Signed-off-by: Prayag Upadhyay <prayag.upd@gmail.com>

Signed-off-by: Prayag Upadhyay <prayag.upd@gmail.com>

prayagupa marked this pull request as ready for review July 15, 2026 23:52

prayagupa requested review from MohammadHaroonAbuomar and liamcrumm as code owners July 15, 2026 23:52

github-actions Bot added documentation Improvements or additions to documentation tests scripts/ci/cd labels Jul 15, 2026

github-actions Bot added the size/XL Extra large PR (500+ lines) label Jul 15, 2026

github-code-quality Bot found potential problems Jul 15, 2026

View reviewed changes

Comment thread tests/e2e_python/scenarios/filesystem_escape/test_filesystem_escape.py Fixed

prayagupa force-pushed the feat/python-governance-e2e branch from 0e12d1f to a05d7d4 Compare July 16, 2026 03:58

liamcrumm requested changes Jul 16, 2026

View reviewed changes

Comment thread tests/e2e_python/scenarios/prompt_injection/test_prompt_injection.py Outdated

Comment thread tests/e2e_python/README.md Outdated

Comment thread tests/e2e_python/README.md Outdated

Comment thread scripts/check_v4_ratchet.py Outdated

prayagupa force-pushed the feat/python-governance-e2e branch from 00e0d69 to b8f8b07 Compare July 17, 2026 18:05

MohammadHaroonAbuomar requested changes Jul 17, 2026

View reviewed changes

prayagupa force-pushed the feat/python-governance-e2e branch from 0884dc6 to ecace4b Compare July 17, 2026 18:54

prayagupa requested review from MohammadHaroonAbuomar and liamcrumm July 17, 2026 19:59

prayagupa force-pushed the feat/python-governance-e2e branch from ecace4b to 88f7a6e Compare July 20, 2026 17:38

prayagupa added 6 commits July 20, 2026 15:41

feat(e2e): add Python governance scenarios and CI workflow

4dee8e9

Signed-off-by: Prayag Upadhyay <prayag.upd@gmail.com>

ci(e2e): run for all non-doc changes

2665479

Signed-off-by: Prayag Upadhyay <prayag.upd@gmail.com>

fix(e2e): clarify filesystem escape scenario

acd550a

Use an explicit protocol stub and a distinct canary value for the blocked filesystem write. Signed-off-by: Prayag Upadhyay <prayag.upd@gmail.com>

fix(e2e): use spell-safe filesystem canary

a93e0eb

Signed-off-by: Prayag Upadhyay <prayag.upd@gmail.com>

replace agent-os policies with ACS policies

d38021d

Signed-off-by: Prayag Upadhyay <prayag.upd@gmail.com>

pin rust-toolchain

c0fec3f

Signed-off-by: Prayag Upadhyay <prayag.upd@gmail.com>

prayagupa force-pushed the feat/python-governance-e2e branch from 88f7a6e to c0fec3f Compare July 20, 2026 22:42

liamcrumm previously approved these changes Jul 20, 2026

View reviewed changes

MohammadHaroonAbuomar and others added 2 commits July 21, 2026 11:31

Merge branch 'main' into feat/python-governance-e2e

4293392

Use the docker daemon's built-in BuildKit

e087f2b

Signed-off-by: Prayag Upadhyay <prayag.upd@gmail.com>

prayagupa dismissed liamcrumm’s stale review via e087f2b July 21, 2026 18:44

ci: add moby and buildkit to cspell dictionary

561b003

Signed-off-by: Prayag Upadhyay <prayag.upd@gmail.com>

MohammadHaroonAbuomar approved these changes Jul 21, 2026

View reviewed changes

MohammadHaroonAbuomar merged commit 829065c into main Jul 21, 2026
135 checks passed

MohammadHaroonAbuomar deleted the feat/python-governance-e2e branch July 21, 2026 19:23

Uh oh!

Conversation

prayagupa commented Jul 15, 2026

Description

Type of Change

Package(s) Affected

Checklist

Attribution & Prior Art

AI Assistance

IP, Patents, and Licensing

Related Issues

Uh oh!

github-actions Bot commented Jul 15, 2026

PR Review Summary

Uh oh!

github-actions Bot commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dependency Review

OpenSSF Scorecard

Scanned Files

Uh oh!

Uh oh!

github-actions Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

API Compatibility

Uh oh!

github-actions Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Docs Sync

Uh oh!

github-actions Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

tests/e2e_python/scenarios/policy_deny/test_policy_deny.py

tests/e2e_python/scenarios/policy_allow/test_policy_allow.py

tests/e2e_python/scenarios/pii_redaction/test_pii_redaction.py

tests/e2e_python/scenarios/prompt_injection/test_prompt_injection.py

tests/e2e_python/scenarios/filesystem_escape/test_filesystem_escape.py

Uh oh!

github-actions Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MohammadHaroonAbuomar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions Bot commented Jul 15, 2026 •

edited

Loading

github-actions Bot commented Jul 16, 2026 •

edited

Loading

github-actions Bot commented Jul 16, 2026 •

edited

Loading

github-actions Bot commented Jul 16, 2026 •

edited

Loading

github-actions Bot commented Jul 16, 2026 •

edited

Loading

`tests/e2e_python/scenarios/policy_deny/test_policy_deny.py`

`tests/e2e_python/scenarios/policy_allow/test_policy_allow.py`

`tests/e2e_python/scenarios/pii_redaction/test_pii_redaction.py`

`tests/e2e_python/scenarios/prompt_injection/test_prompt_injection.py`

`tests/e2e_python/scenarios/filesystem_escape/test_filesystem_escape.py`

github-actions Bot commented Jul 16, 2026 •

edited

Loading